1.1 Logit agents

Logit agents use the discrete choice model fit from the paper to pick which statistical area to trawl next.
More precisely, given a vector of logit coefficients \(\beta\) and the design matrix \(X\) where each row \(x_i\) contains the observations regarding statistical area \(i\), the probability of fishing in statistical area \(i\) is: \[ \text{Pr}(i) = \frac{e^{x_i \beta}}{ \sum_{j} e^{x_j \beta}} \] Once a statistical area is picked, a cell at random within that area is trawled

Expressed in pseudo-R, logit agents act as follows:

# Given K statistical areas
# Logit coefficients beta
# And observations (distance, habit and intercepts) for each area 

#compute utility (logits)
for(statistical_area in 1:K)
  utility[statistical_area] = beta*observations[statistical_area,]

# Use SOFTMAX to pick one statistical area:
# probability of choosing arm i is proportional to the exp of its utility
denominator<- sum(exp(utility))
probabilities<- exp(utility)/denominator
# choose statistical area based on that probability
statistical_area<-sample(1:K,prob = probabilities,size = 1)
# target a random cell in that statistical area
target<-sample(cells[statistical_area],size=1)

This is the standard way to simulate historical behaviour in fishery agent-based models (see for example Saul, 2012; Rose et al., 2015). Because its \(\beta\) vector comes from a statistical fit, it requires no further calibration.

1.2 RUM Agents

RUM agents function just like logit agents but:

1. They use a larger set of independent variables \(X\)
2. Their parameters were tuned to minimize outcome error within POSEIDON

For these agents the \(x\) vector in the logit probability is made up of the following:

1. Distance (in km) from homeport
2. Habit: number of times that location was fished in the past 365 days
3. Revenue/hr: 365 days moving average of revenues made at that location (ignoring all costs) divided by length of trips in hours
4. CPUE of
   • sablefish
   • Dover sole
   • yelloweye

There is a global intercept which is necessary for agents not to get stuck at the very first area they trawl.

There are two variants of the RUM agent: RUM fleetwide where the Revenue/hr \(x\) averages across the experience of the whole fleet rather than individual observations and RUM precise where the agents perform the logit selection over the finer POSEIDON cells rather than the statistical areas.

1.3 Historical agents

Historical agents are a less elegant (but more effective) way to force agents to act historically. Agents only consider statistical areas within 200km of their home port. They target each area with probability proportional to how often that area was visited between 2011 to 2014. For example, if the agent has to choose between statistical area A and B, and the logbook data shows that area A was visited 900 times while area B 100 times then the agent will visit A with probability \(\frac{9}{10}\).

Expressed more precisely in algorithm form:

# Given K statistical areas
# given trips[] vector containing empirically observed visits per statistical area

#ignore all areas that are too far
for(statistical_area in 1:K)
  if(distance(statistical_area) > 200)
     trips[statistical_area] = 0

# normalize trips into probabilities
trips<- trips/sum(trips)
# choose statistical area based on that probability
statistical_area<-sample(1:K,prob = trips,size = 1)
# target a random cell in that statistical area
target<-sample(cells[statistical_area],size=1)

If we judge statistical accuracy in terms of how much each statistical area was visited between 2011 and 2014, then the historical agent is more accurate than the logit one. This is because logit agents have a high \(\beta\) associated with “habit” variable and tend to often get stuck in suboptimal areas. When we judge statistical accuracy by random-utility choice fits, then the logit agents are more accurate than the historical ones.
Historical agents then look more accurate in aggregate but less so when studying their individual dynamics.

1.4 Bandit agents

The multi-armed bandit problem (Bubeck & Cesa-Bianchi, 2012; Kuleshov & Precup, 2014; Vermorel & Mohri, 2005) is a classic model of the exploration-exploitation tradeoff. In its original form an agent faces a set of slot machines; each slot machine’s payoff is stochastic but some machines are better than others. The agent can only play one slot machine at a time and has to balance the need to try new slot machines to learn their payoffs with the cost of not playing the slot machine that so far has proven superior.
Many heuristics, bandit algorithms, have been proposed to solve this class of problem but here we focus only on the simplest one: the \(\epsilon\) greedy bandit.

Each trip the agent has probability \(\epsilon\) of trying a statistical area at random. If not the agent will go back fishing the area whose average profits have been highest so far. At the end of each trip, the agent observes the profits made and updates its valuation of the statistical area visited. More precisely, the algorithm proceeds as follows:

# Given K statistical areas
# given experience[] vector containing average profits observed so far by the agent

# with probability epsilon, make random choice
if(runif() < epsilon)
    chosen_area <- sample(1:K,size=1)
# otherwise pick the area that so far made the most profits
else
    chosen_area<-which.max(experience)  
    
target<-sample(cells[chosen_area],size=1)
    
# at the end of the trip, update your experience by 
# exponential moving average
experience[chosen_area]<- alpha * experience[chosen_area] + (1-alpha) * profits_made

The algorithm has 2 parameters: the exploration rate \(\epsilon\) and the exponential moving average’s smoothing parameter \(\alpha\).

1.5 Explore-Exploit-Imitate agents

Explore-exploit-imitate is the default algorithm in POSEIDON. Much like the \(\epsilon\)-greedy algorithm, it has a fixed probability of exploring a new cell (although in this instance the exploration is in the neighborhood of the last cell visited, not a random new statistical area).
When not exploring, the agent has a probability of imitating a “friend” who is making more money. Otherwise the fisher targets the last cell trawled.

Each fisher has 2 “friends” she keeps track of. Each fisher knows where their friends last trip was and how much money it made. Friendship is only possible between boats of the same port. Friendships are not necessarilly reciprocal.

In pseudo-R the explore-exploit-imitate algorithm looks as follows:

# with probability epsilon, explore
if(runif()<epsilon)
  # pick a new area at random in the neighborhood of your last cell visited
  # where delta is the exploration range
  options<- von_neumann_neighborhood(last_cell_visited,
                                     delta)
  #go there
  target<-sample(options,size=1)
else
  # if a friend is doing better, target their area with probability gamma
  if(runif < gamma && profits[me] < profits[best_friend])
    target<- position[best_friend]
  # otherwise go back to the same cell
  else
    target<-last_cell_visited


#if explored and the profits are worse than before, then go back
if(profits(last_cell_visited)<profits(cell_visited_two_trips_ago))
  last_cell_visited<-cell_visited_two_trips_ago

The algorithm depends on 3 parameters: the exploration rate \(\epsilon\), the exploration range \(\delta\) and the imitation probability \(\gamma\).

1.6 Social annealing agents

Social annealing agents are a slight variation over EEI where they do not know where other fishers trawl but they do know the profits others are making. Whenever a social annealing agent is making \(k\)% worse than average fishery profits, she explores a new cell at random. Otherwise she targets to the last cell visited.

More precisely, the algorithm proceeds as follows:

# knowing last_cell_visited as the cell targeted in last trip

# if you are making less than k times the average
if(profit_made < k * average_profits)
  # pick a new area at random in the neighborhood of your last cell visited
  # where delta is the exploration range
  options<- von_neumann_neighborhood(last_cell_visited,
                                     delta)
  #go there
  target<-sample(options,size=1)
else
  #otherwise keep fishing in the same spot
  target<-last_cell_visited

The algorithm depends on two parameters: the threshold parameter \(k\) and the exploration range (in map cells) \(\delta\). This algorithm first appeared (with \(\delta=\infty\)), to the best of our knowledge, in Beecham & Engelhard (2007).

1.7 Heat-mapping agents

Heat-mapping agents build a statistical model of profit per cell (the eponymous heatmaps) and update it and the end of each trip. With a fixed probability \(1-\epsilon\) they target the peak of the heatmap otherwise fishing in its neighborhood.

More precisely the heat-mapping agent builds a statistical heatmap by using a kernel regression where for each cell \(x\) we initialize variables \(g_0,m_0=0\).
Whenever the fisher observes profits \(\Pi_y\) at cell \(y\), it updates its estimate profitability of cell \(x\) by: \[ \begin{aligned} g_n &= g_{n-1} + K(x,y) \\ m_n &= m_{n-1} + \frac{ \left(\Pi_y - m_{n-1} \right) K(x,y)}{g_n} \end{aligned} \] Where \(K(\cdot,\cdot)\) is a kernel function and \(m_n\) is the predicted profitability of \(x\).
To give more weight to more recent observations we add a forgetting factor \(0<\lambda<1\) as follows: \[ g_n = \lambda g_{n-1} + K(x,x_n) \] We use here as kernel function: \[ K(x,y) = exp(-\frac{\text{Distance}(x,y)^2}{2\sigma}) \] Where \(\sigma\) is the space bandwitdh (that is, it modulates how much far away observations affect profitability predictions here).
This recursive version of the Kernel regression first appeared in Krzyzak (1992).

Given the statistical model, the heat-mapping agents proceed as follows:

# given heatmap_predictions: the predictions made by the fisher about each cell profitability
best_cell<-which.max(heatmap_predictions)
# with probability epsilon, explore around the best_cell
if(runif()<epsilon)
  # where delta is the exploration range
  options<- von_neumann_neighborhood(best_cell,
                                     delta)
  #go there
  target<-sample(options,size=1)
else
  # pick the top
  target<-best_cell

It is the strategy with more parameters to calibrate: \(\delta\) is the exploration range, \(\epsilon\) is the exploration probability, \(\sigma\) is the space bandwitdh, \(\lambda\) is the forgetting factor.

1.8 Random agents

Random agent simply pick a cell at random each trip. It is a very poor strategy but it is important to use as a baseline: if other algorithms’ performance equals the random agents’ one then those algorithms aren’t useful. They have no parameters to calibrate

1.9 Perfect SA agents

Perfect-SA agents pick a random cell for each statistical area, compute the profitability for each of those cells and pick one in proportion to the exponential of the expected profits (like logit agents). The agent has perfect knowledge in the sense that it knows ahead of time the profits she will make for each cell. However the agent does not update its computation halfway through a trip. This means that if multiple perfect agents target the same cell, their profits might be lower than expected at the beginning of the trip. These agents do not have the infinite computational power needed to analyse the game theoretical implications of everyone else being as informed as they do.

More precisely, perfect agents act as follows:

# given K statistical areas

#pick a cell for each area
for(statistical_area in 1:K)
  cell[statistical_area]<-sample(cells[statistical_area],size=1)
  #compute profits for each cell
  profits[statistical_area]<-forecast_profits(cell)

# Use SOFTMAX to pick one statistical area
denominator<- sum(exp(profits))
# probability of choosing arm i is proportional to the exp of its utility
probabilities<- exp(profits)/denominator
# choose statistical area based on that probability
statistical_area<-sample(1:K,prob = probabilities,size = 1)
# target selected cell in that statistical area
target<-cell[statistical_area]

1.10 Perfect Cell Agents

Perfect cell agents just pick the POSEIDON cell that makes the most profit knowing ahead of time how much fish is there in each cell. Like the Perfect SA agent it doesn’t take into consideration other fishers and their actions. Unlike Perfect SA it does not use logit to make its decision stochastic.

2.1 Individual Errors

2.2 Qualititative Validation

Holland (2016) tabulates the quota prices between 2011 and 2014 observed in the groundfish fishery. We can compare this data with the quota prices generated from our model. Holland however reports that a large amount of quotas are bartered rather than traded and that prices vary much during the year: only 18% of quota transfers in 2011 involved cash.

We look at the price of quotas traded within the simulation and compare them with Holland’s numbers. Fundamentally, the constraining species in the model is sablefish whose quota is most expensive; the better the algorithm is at fishing, the more sablefish ends up costing. Other quotas are rarely traded and if so they are at the minimum price (0.05); this is true also for yelloweye which is so rare that is simply not traded as nobody really form high expectations of catching them.

Figure 2.1: Five sets of box plots, one for each species’ quota prices. The box plots show the yearly average trade price over 100 simulated runs. The red line is the 2011-2014 average as in table 2 of Holland (2016)

As a second test we plot average daily profits over the years in figure 2.2. Data on profit trends from the FISHeries Economics Explorer dataset (Northwest Fisheries Science Center, 2017) shows a peak in variable costs net revenue in 2011, then a slight decline until another peak in 2015. However the confidence intervals on the time series (with standard deviation approximately equal to the mean) are much larger than the trend itself which make any inference and calibration suspicious. Our main focus here was to make sure there was movement of more than 2 or 3 times the mean itself.

Figure 2.2: The simulated profits time series, one for each adaptive decision-making algorithm

As a third test we plot the number of fishers left in the simulation at the end of each year in figure 2.3. From the catcher vessel report we know the number of fishers remaining in the DTS fishery (Steiner, Pfeiffer, Harley, Guldin, & Lee, 2016), however these numbers change between each revision of the dataset. Moreover each year about 10-15% of boats in the West-Coast groundfish fishery make operating losses so that our simulated quitting decision (making two consecutive years of losses) might be overly stringent. Still, most simulations settle around the right number of fishers.

Figure 2.3: Number of active fishers left in each simulation each year. The dashed red line shows the number of non-whiting DTS boats left by 2015.

Finally, we plot in figure 2.4 the active fishers per port at the beginning and at the end of the simulation when using explore-exploit-imitate agents. The geographical allocation of other adaptive algorithms is similar. The model tends to predict too few fishers in Astoria and too many in Newport and Coo’s Bay. The model however does predict the decrease in DTS participation in Washington and is also accurate in California.

Figure 2.4: Box-plot of active fishers per port (ordered from north to south along the coast) when the decision-making algorithm is the calibrated explore-exploit-imitate over 100 simulated runs. The red squares in the 2014 plot are the active fishers per port reported by Steiner et al. (2016)

3.1 Pre-2011 Regulations

Because our model runs only for a short amount of time, we incur the risk of calibrating the model too tightly around the few years for which we have data. In this section we test for this risk by starting the model in 2007 rather than 2011. This involves changing starting abundance for each species, different gas prices, variable costs and implementing the 2011 regulatory shock when a different set of quotas were introduced and made tradeable. We run the model again until 2015 without recalibrating it. We test its prediction of both the 2011-2014 period (calibration error) and 2015 (validation error).

Before 2011, the fishery was regulated through bi-monthly trip limits (National Marine Fisheries Service, 2009). These rules changed frequently, were sometimes superseded by state regulations and varied between states. We did not try to model these accurately as we used no pre-2011 data to calibrate or validate the model. Our main focus was to have a different regime policy our agents would transition away from.

We follow again Toft, Punt, & Little (2011) here in simplifying these regulations as individual non-tradeable quotas. We need to do so because we do not simulate explicitly fishing seasons and weather patterns. Table 3.1 shows the aggregate quota numbers we derived. All quotas are higher except for Dover sole; however because Dover sole attainment rate is low, the overall effect of using pre-2011 quotas is more total landings.

Figure 3.1: Box plot showing the fishing outcomes error for 100 simulations of each decision algorithm. All runs started in 2007 and transitioned to ITQ in 2011

All the results from validation are upheld. Adaptive agents do better, statistical agents underperform in 2015, perfect agents catch too much yelloweye and finally bandit and social annealers fit worse than in the calibration phase.
The main difference is that heatmap agents now do better than EEI.

3.2 CPUE map

In this section we change the distribution of fish while keeping the overall number and stock assessment-driven dynamics constant. We show that the model is sensitive to this change and that the main driver is the degree of smoothness by which fish is initially distributed.

Originally we distributed fish according to the Essential Fish Habitat statistical models; here we distribute fish proportional to the CPUE (catches per unit of effort) recorded in the logbook data for each statistical area.
Branch et al. (2006) provides plenty of reasons not to use CPUE as an abundance metric and we only do so for sensitivity analysis.
Logbook observations are sparse so we apply a linear interpolator to have a CPUE prediction for each cell of the map.

The next figure shows how the abundance of sablefish changes as we change data source. Because of the linear interpolation, the CPUE-derived map distributes fish much more smoothly.

Figure 3.2: Image capture of abundance for sablefish in the model as derived from the EFH dataset (on the left) and the CPUE interpolation (on the right)

Figure 3.3 shows the errors obtained for each decision-making algorithm when calibrated against the new distribution of fish. Three results emerge. First, the error in absolute terms is much higher for all algorithms. Second, statistical algorithms outperform adaptive algorithms. Third, perfect agents perform as well as adaptive algorithms. The only result that survives this sensitivity analysis is that random agents still perform very poorly.

Figure 3.3: The calibration errors when using the CPUE map, each box plot represents 100 simulation runs.

All these results are due to the smoothness of geographical abundance. Figure 3.4 shows the profit simulated for each decision-making algorithm. The figure makes clear that it is not the profit maximizer agent that got more realistic, it is adaptive agents that now also make too much money. Smooth abundance distribution generates smooth profit curves that are easy for all adaptive agents to climb. When the problem is easy, assuming adaptive agents is like assuming perfect agents. Statistical agents perform well because they do not take advantange of the easier problem: they are forced to repeat historical choices even if better alternatives are present.

Figure 3.4: Yearly profits made by each decision-making algorithm in 100 simulated runs. Red dashed line is the empirical 2011-2014 average

It is instructive to look in detail at the consequences of using CPUE-driven maps. One subtle property of the map is shortspine and sablefish being almost perfectly co-located. This has less to do with the two species being actually co-located and instead simply represents the efficiency of fishermen in Astoria (a good reason for not using CPUE-driven maps). The model reacts in a realistic way: because sablefish is the target species and it is now impossible landing it without catching shortspine as well, shortspine quotas become more expensive; as we show in figure 3.5 quotas are sometimes even traded at prices higher than shortspine’s worth, meaning that agents take a loss by landing shortspine just so that they can land sablefish as well.

Figure 3.5: Shortspine thornyhead quota prices traded for each simulation using either EFH or CPUE abundance maps. The red dashed line represents the empirical quota price observed, the dashed black line represents the ex-vessel price. With the CPUE-driven map we see not only quotas more expensive than what was empirically observed but sometimes more expensive than their sale prices

It is not possible to speculate whether CPUE maps are always a poor idea for agent-based models. In this fishery effort is clustered near the coast and around some key Oregon ports, while the EFH allocates fish further out and more homogeneously. The issues with linear interpolations can also be prevented by higher resolution observations.

3.3 Ignore boats making only one trip

One problem with computing profits by averaging over boats is that there are a few agents who only make one trip each season filling only a limited amount of quota. The profits made by those boats are small and they tend to lower the average profit made by the fishery.
We test this here by recalibrating the model to observe only profits of boats making more than one trip a season. This shuffles the ranking of adaptive agents as now heatmap agents do better than EEI, keeping most of the rest of the results constant. The one major difference is that ranking by elastic nets in this instance prefers historical agents to adaptive ones.

3.4 Unified sablefish market

In the model we remove quotas that eventually were landed by other gears. This is accurate but had we truly looked into the future rather than hindcasting we would have not had that information. Would behavioural ranking change if we made north and south sablefish quota all available to the DTS boats? Behavioural ranking does not change but errors are in general all higher due to too much sablefish being landed.

4.1 Historical

We first perform an active non-linear test to the calibrated catchability (\(Q\)) and hold-size (\(H\)) parameters by varying them 10% around their optimal value with historical agents. We show the density plot of fishing outcome error in figure 4.1. The search increases the error by only 11% on average. While the difference is significant to a Wilcox test, it is linearly proportional to the 10% shock in parameters.

Figure 4.1: The histogram of fishing outcome errors generated by 100 runs of the step 1 calibrated model (that is, using historical agents) at the bottom versus the worst performing set of parameters found by the active non-linear test.

4.2 EEI

Explore-exploit-imitate agents are very insensitive to parameter changes. In figure 4.2 we compare the calibrated and worst possible parameter sets (changing behavioural parameters by at most 10%). The error is statistically significant to a Wilcoxon test, but the overall increase is only of 6%. Even though the explore-exploit-imitate sensitivity analysis has more parameters it is actually stabler the historical agents used above.

Figure 4.2: The density plot of fishing outcome errors generated by 100 runs of the step 2 calibrated model using explore-exploit-imitate at the bottom versus the worst performing set of parameters found by the active non-linear test.

4.3 Kernel

Heat-mapping agents are more sensitive to active non-linear tests (particularly to changes in forgetting factor \(\lambda\)). As shown in figure 4.3 the average fishing outcome error increases only by 13% but the likelihood of large errors is higher. In a way this is stating the obvious: heat-mapping agents are more complex than explore-exploit-imitate ones and we thus expect them to be more delicate to parameter changes.

Figure 4.3: The density plot of fishing outcome errors generated by 100 runs of the step 2 calibrated model using heatmap agents at the bottom versus the worst performing set of parameters found by the active non-linear test.

4.4 Movement Test

From Alaskan tagging data we know that sablefish moves on average 191 km per year (Hanselman, Heifetz, Echave, Dressel, & Jech, 2015). We also know from simple abstract model runs that fish diffusion rates in POSEIDON have large effects on long term profitability and behaviour of the simulated fleet. For these reasons we initially classed sablefish movement as a free parameter to calibrate much like catchabilities and hold-sizes. In these early calibration runs however movement rate did not have a significant effect in minimizing distance from data (which in a Bayesian optimization can be noticed by the very large bandwitdhs assigned to the variable by the optimiser or just by plotting posteriors). In the final version of the calibration presented above the movement rate was simply set to 0 as it made the model much faster to run.

Similar to Goethel et al. (2011), we modeled movement as a simple diffusion rate between cells. However, we biased movement in such a way that it would never alter the patchiness of the original map. Movement in this simulation is not from cells where there are more fish to cells where there are a few; rather fish moves from “full” areas to “empty” ones where full and empty are defined in terms initial abundance distribution. To make a concrete example, take cell A and a contiguous cell B where cell A contained \(k_a\) percent of the total sablefish at the beginning of the simulation and cell B contained \(k_b\) percent. If there currently are \(f_a\) sablefish in cell A and \(f_b\) in cell B, then fish movement between A and B is given by: \[ m \frac{f_a k_b - f_b k_a}{k_a+k_b} \] where \(m \in [0,1]\) is the diffusion rate parameter. This means that even at maximum movement rate distribution of fish remains as patchy.

Figure 4.4 shows the difference in error generated by maximizing sablefish diffusion rate. Validation error is fundamentally unchanged. Calibration error (not shown) increases by 9%. While we know that POSEIDON is sensitive to movement rates in other scenarios here we showed that when simulating short periods of time there is little error added by ignoring movement and it is anyway hard to back out a proper movement rate from data without longer observations.

Figure 4.4: Density plot of validation error generated by 100 runs with movement turned off and with movement turned on, \(m=0.1\)

5.1 Validation Minimizing RUM

RUM agents perform poorly in the validation section primarily by overfitting. Would they have been much different if we had “cheated” and calibrated them against validation error? Statistical area agents (RUM and RUM SA) see almost no improvement but RUM precise agents can halve their validation error if reparametrized. Figure 5.1 shows the difference in calibration and validation error distribution for the original RUM precise agent (whose parameters minimize calibration error) and the “cheating” RUM precise agent (whose parameters minimize validation error).

Figure 5.1: Empirical densities of validation and calibration error for RUM precise agents. As expected the cheating agent sacrifices fit quality in the calibration phase in order to do better in validation.

The key result of this exercise is to compare the optimal parameters for the original RUM precise and the one that minimizes validation error instead:

The main change is the difference in habit which goes from very positive to very negative (compare this to the 0.21 coefficient in the logbook-derived Logit agent). This is introducing purposeful exploration “by the backdoor”: it is a way to keep the agent changing its destination even if the other indicators do not change.

This exploration makes a difference in even the few years that are simulated as figure 5.2 shows: profits decrease much faster for the original RUM Precise than the cheating one which are pushed to move around over time.

Figure 5.2: Empirical densities of validation and calibration error for RUM precise agents. As expected the cheating agent sacrifices fit quality in the calibration phase in order to do better in validation.

5.2 Validation Minimizing Social Annealing

Social annealers perform poorly because the parameters for which they are calibrated (\(k=0.548\) and \(\delta=1\)) generate very easily satisfied agents who have little reason to explore and improve fishing efficiency. This results in a steady decline of catches, profits and eventually number of active fishers.

If we “cheat” and look instead for the parameters that minimize validation error we see in figure 5.3 that it is possible for social annealers to produce low validation errors but in doing so we abandon the “satisficing” assumption that underpins this agents. This is because the parameters that minimize validation error have agents that are deeply unsatisfied (\(k=2.64\) means that they will keep exploring until they make at least 264% more than the average fisher) and quite risk prone (\(\delta=19\) implies a very wide exploration range). At least for this formulation of the satisficing principle, the only way to fit it to data is to twist enough that it generates hypercompetitive behaviour nonetheless.

Figure 5.3: Empirical densities of validation and calibration error for RUM precise agents. As expected the cheating agent sacrifices fit quality in the calibration phase in order to do better in validation.